Papers with English benchmarks
Multilingual Email Zoning (2021.eacl-srw)
Copied to clipboard
| Challenge: | Existing literature on email zoning is mainly limited to English . however, it is possible to discern a level of formal organization in the way most emails are formed. |
| Approach: | They propose a multilingual email zoning benchmark based on a language agnostic sentence encoder and a new model that uses a biLSTM with a CRF to classify each sentence into an email zone. |
| Outcome: | The proposed model is competitive with current English benchmarks and reached state-of-the-art performance in English. |
GPTAraEval: A Comprehensive Evaluation of ChatGPT on Arabic NLP (2023.emnlp-main)
Copied to clipboard
| Challenge: | Our study examines ChatGPT’s performance on Arabic languages and dialectal varieties. |
| Approach: | They conduct a large-scale automated and human evaluation of ChatGPT, encompassing 44 distinct language understanding and generation tasks on over 60 different datasets. |
| Outcome: | The proposed model outperforms smaller models on Arabic dialects compared to GPT-4's Modern Standard Arabic and Dialectal Arabic (DA) |
Discrete Opinion Tree Induction for Aspect-based Sentiment Analysis (2022.acl-long)
Copied to clipboard
| Challenge: | Dependency trees are used for aspect-based sentiment classification but are not optimized for aspect classification. |
| Approach: | They propose an aspect-specific and language-agnostic discrete latent opinion tree model as an alternative structure to explicit dependency trees. |
| Outcome: | The proposed model can achieve competitive performance and interpretability on six English benchmarks and one Chinese dataset. |
KMMLU: Measuring Massive Multitask Language Understanding in Korean (2025.naacl-long)
Copied to clipboard
Guijin Son, Hanwool Lee, Sungdong Kim, Seungone Kim, Niklas Muennighoff, Taekyoon Choi, Cheonbok Park, Kang Min Yoo, Stella Biderman
| Challenge: | Recent models struggle to show performance over 60%, significantly below the pass mark of the source exams (80%), highlighting the room for improvement. |
| Approach: | They propose to use Korean exams to collect 35,030 questions from an expert-level multiple choice model to capture linguistic and cultural aspects of the Korean language. |
| Outcome: | The proposed benchmark is based on 35,030 questions from original Korean exams. |
Exploring a Unified Sequence-To-Sequence Transformer for Medical Product Safety Monitoring in Social Media (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Adverse Events (AEs) are harmful events resulting from the use of medical products. |
| Approach: | They propose a model that combines sequence-to-sequence learning with language transfer capabilities to improve model robustness. |
| Outcome: | The proposed approach achieves strong performance over baselines on English benchmarks. |
Candidate-Aware Retrieval and Reranking for Multiple-Choice Question Answering: Arabic as a Case Study (2026.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) have recently achieved impressive results on multiple-choice question answering (MCQA) despite advances in English, LLMs continue to underperform in Arabic due to gaps in data coverage, linguistic transfer, and evaluation design. |
| Approach: | They propose a method that jointly models the relevance of both the question and its candidate answers when selecting contextual passages. |
| Outcome: | The proposed approach outperforms standard RAG baselines and reranker baselines while remaining competitive with considerably larger models. |
INJONGO: A Multicultural Intent Detection and Slot-filling Dataset for 16 African Languages (2025.acl-long)
Copied to clipboard
Hao Yu, Jesujoba Oluwadara Alabi, Andiswa Bukula, Jian Yun Zhuang, En-Shiun Annie Lee, Tadesse Kebede Guge, Israel Abebe Azime, Happy Buzaaba, Blessing Kudzaishe Sibanda, Godson Koffi Kalipe, Jonathan Mukiibi, Salomon Kabongo Kabenamualu, Mmasibidi Setaka, Lolwethu Ndolela, Nkiruka Odu, Rooweither Mabuya, Shamsuddeen Hassan Muhammad, Salomey Osei, Sokhar Samb, Dietrich Klakow, David Ifeoluwa Adelani
| Challenge: | Slot-filling and intent detection tasks are well-established tasks in Conversational AI, but current benchmarks for these tasks rely on evaluations of low-resource languages and translations from English benchmarks. |
| Approach: | They propose to use a multilingual, open-source benchmark dataset for 16 African languages with utterances generated by native speakers across diverse domains. |
| Outcome: | The proposed dataset compares multilingual transformer models and prompting large language models (LLMs) with the English language. |
TF-LM: TensorFlow-based Language Modeling Toolkit (L18-1)
Copied to clipboard
| Challenge: | Existing deep learning tools offer building blocks but training and building models takes time and knowledge. |
| Approach: | They propose to make available LSTM language models trained on Dutch texts and English benchmarks. |
| Outcome: | The proposed model can be used to test the perplexity, predict the next word(s), re-score hypotheses or generate debugging files for interpolation with n-gram models. |
Span-based Semantic Role Labeling as Lexicalized Constituency Tree Parsing (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing models for semantic role labeling fail to capture the relationship between syntax and semantics. |
| Approach: | They propose a lexicalized tree representation for span-based SRL that integrates constituency and dependency parsing to explicitly model predicate-argument structures. |
| Outcome: | The proposed model achieves competitive performance on standard English benchmarks. |
A dynamic programming algorithm for span-based nested named-entity recognition in O(n2) (2023.acl-long)
Copied to clipboard
| Challenge: | Named entity recognition (NER) is a fundamental problem in information retrieval . nested NER has a cubic-time complexity, but can be realized in quadratic time using a semi-Markov model . |
| Approach: | They propose a span-based named-entity recognition algorithm with a quadratic-time complexity . they add a constraint on the search space to reduce the complexity of the algorithm . |
| Outcome: | The proposed algorithm covers a large part of three standard English benchmarks and delivers comparable results. |
KBioXLM: A Knowledge-anchored Biomedical Multilingual Pretrained Language Model (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing models for multilingual biomedical training are monolingual, resulting in limited cross-lingual capability. |
| Approach: | They propose a model that transforms a multilingual biomedical corpus into a biomedically domain using a knowledge-anchored approach. |
| Outcome: | The proposed model outperforms monolingual and multilingual models in cross-lingual scenarios. |
Automatically Generated Definitions and their utility for Modeling Word Meaning (2024.emnlp-main)
Copied to clipboard
| Challenge: | Modern language models generate semantic representations for words based on context and context based models. |
| Approach: | They propose to use dictionary-like sense definitions to generate sentence embeddings . they evaluate the quality of the generated definitions on existing English benchmarks based on the results of their study . |
| Outcome: | The proposed model sets new state-of-the-art results on lexical semantics tasks compared to baselines . |
Targeted Syntactic Evaluation for Grammatical Error Correction (2025.acl-long)
Copied to clipboard
| Challenge: | Existing evaluation datasets based on learner-produced texts are insufficient for evaluating models . Currently, sequence-to-sequence models and sequence tagging models perform well on beginner-level grammar items . |
| Approach: | They propose a new evaluation paradigm that assesses GEC models using minimal pairs of ungrammatical and grammatically paired sentences for each grammar item. |
| Outcome: | The proposed evaluation paradigm assesses models using minimal pairs of ungrammatical and grammatically-spaced sentences for each grammar item. |